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ABSTRACT 

S. Jackson and S. Jacobs' article entitled 
"Generalizing about Messages" represents an important methodological 
innovation in communication research* The article urges procedural 
change by recommending the use of multiple-message designs because of 
a weakness in single-message designs; it recommends analyzing 
message-replication data with a random-effects analysis because such 
a procedure provides the appropriate basis for the generalizations 
that are wanted* That is, "Generalizing about Messages" proposes both 
a certain sort of data-gather ing procedural change (i.e*, message 
replications) and a data~analyt ic procedural change (i.e*, 
random-effects analysis). Indeed, "Generalizing about Messages" has 
significantly changed customary research procedures in communication* 
It is much more common now to see multiple message designs* 
"Generalizing about Messages" is unique in its exploration of the 
interconnection of procedure and substance — it does not separate 
issues of methodology from issues of substance. Its implications are 
far-reaching* For instance, the same arguments underwriting the use 
of random-effects analyses in primary research designs with multiple 
messages also underwrite the use of random-effects analyses in 
meta-analysis* Broadly put, the relevant general principle is that 
replications should be treated as random whenever the underlying 
interest is in generalization. The article also invites consideration 
of the variability of effect on a phenomenon* (Contains eight 
references.) (TB) 
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Methodological Innovation in Communication Research 



"Generalizing About Messages" (Jackson & Jacobs, 1983) represents an 
important methodological innovation in communication research, and I want to 
take this opportunity to reflect on the nature of such innovation. 

Methodological innovation makes for a change in our research procedures. 
Broadly put, this commonly takes the form of the use of a new tool, as when 
some new data-gathering device takes shape (as in think-aloud protocols) or 
when some new data-analyt ic procedure appears on the scene (e.g. , canonical 
correlation) . Procedural change is (naturally enough) the heart of 
methodological innovation. 

And so it is with "Generalizing About Messages." This is a 
methodologically focussed paper that urges procedural change: it recommends 

the use of multiple-message designs, because of the weaknesses of single- 
message designs; and it recommends analyzing message-replication data with a 
random-effects analysis, because such a procedure provides the appropriate 
basis for the generalizations that are wanted. That is, "Generalizing About 
Messages" proposes both a certain sort of data-gathering procedural change 
(viz., message replications) and a data-analyt ic procedural change (viz., 
random-effects analysis) . 

And, in fact, "Generalizing About Messages" has significantly changed 
customary research procedures in communication (see Brashers, 1996). It's much 
more common now to see multiple-message designs; when single-message are used, 
often they're accompanied by an acknowledgement of the limitations of the 
design; and so forth. 

This, by itself, makes "Generalizing About Messages" a worthy recipient 
of the Woolbert award. After all, a research community not only accumulates 
findings and theories, but also builds up a storehouse of knowledge about 
research methods. "Generalizing About Messages" has made an important 
contribution to that fund of methodological knowledge, by leading to 
significant procedural change. 

But this way of thinking about methodological innovation — thinking about 
it as procedural change — is in some ways misleading, because it separates 
procedure from substance. To be sure, methodological training commonly 
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enforces such a separation. For one thing, graduate-program requirements 
commonly contain a separate "methods" requirement, as something different from 
substantive coursework. And communication students are often sent off to other 
departments to get their "methods" training. In the case of statistical 
training, the nature of statistical procedures suggests such a distinction, 
since the procedures are (largely) indifferent to substance: an average is an 

average, no matter the substantive nature of the variable under examination. 
That's why so much of statistics can be learned, at least initially, outside of 
the student's substantive context of interest. (Statistics is not unique in 
this regard: the same thing is true in the use of foreign languages as 

"tools.") So the separation of method from substance is in some ways entirely 
understandable . 

But what I want to underscore here is the interconnection of procedure 
and substance, as represented precisely by "Generalizing About Messages." And 
as an initial observation, notice that "Generalizing About Messages" is very 
much a message-centered paper, one focused on message-centered methodological 
innovation. What I mean is that this is not a discussion of some general 
abstract methodological innovation (of the sort represented by the development 
of some broad new statistical procedures, along the lines of LISREL or 
suchlike) . This paper is instead focussed specifically on the problem of 
generalizing about messages . 

And it approaches the problem of generalizing about messages as a 
distinctive one. For example, the arguments it invokes to underwrite the 
importance of replications explicitly consider the nature of the objects under 
investigation: "We have no theories of language and communication rich enough 

to seriously attempt exhaustive analysis of any particular case of 
communication (even granting the possibility of an exhaustive description) , so 
we have no means by which to bring under control all the unwanted effects of 
language" (Jackson & Jacobs, 1983, p. 171) . Or: "Elements in a linguistic 

unit may function differently in combination with each other in one context 
than they do in other contexts" (p. 171) . That is to say, the recommended 
procedural changes are prompted by consideration of the nature of communication 
phenomena. The heart of the paper is thus specifically the problem of adapting 
procedures to fit interests in message generalization. 
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Now given the article's focus on generalization across multiple messages, 
one thing that's striking (in re-reading the article) is that it makes no 
mention of mata-analysis . Of course, in 1983, meta-analysis was pretty much a 
figure on the horizon in communication research (and elsewhere, for that 
matter). Hunter, Schmidt, and Jackson's (1982) book had only just appeared; 
and (what I think may be) the first meta-analysis in communication was 
presented at ICA in 1983: Dillard, Hunter, and Burgoon's (1983) review of the 

foot-in- the-door and door- in- the- face literatures. Perhaps, then, it's 
unsurprising that "Generalizing About Messages" makes no mention of meta- 
analysis . 

But there is obviously a parallel concern with the use of replications as 
a better basis for generalizing. Notably, one of the ways in which this work 
has been extended is precisely the consideration of the relationship between 
the analysis of primary-research multiple-message designs (as discussed in 
"Generalizing About Messages") and the meta-analytic treatment of data gathered 
from replicated single-message designs. It's become clear that there is in 
fact an underlying similarity between (for example) ANOVA treatment of a 
replicated primary research design and the parallel meta-analytic treatment of 
such data (see Jackson, 1992, pp. 118-123). 

What perhaps has not yet been so widely appreciated, however, is that the 
same arguments underwriting the use of random-effects analyses in primary 
research designs with multiple messages also underwrite the use of random- 
effects analyses in meta-analysis. Broadly put, the relevant general principle 
is that replications should be treated as random whenever the underlying 
interest is in generalization. This principle reflects the fact that fixed- 
effects and random-effects analyses test different hypotheses. For instance, 
when comparing two group means while treating message replications as fixed, 
the hypothesis that is tested concerns whether the responses to a fixed, 
concrete group of messages differ from the responses to some other fixed, 
concrete group of messages; the parallel random-effects analysis tests whether 
responses to one category of messages differs from responses to another 
category of messages (see, e.g. , Jackson, 1992, p. 110). 

This is of some importance, because the default option in most meta- 
analytic work in communication (and elsewhere) has been fixed-effects analyses. 
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even though investigators are typically interested in generalizing beyond the 
cases at hand. Perhaps it is unsurprising that a National Research Council 
panel should have concl ude d that meta-analytic work "would be improved by the 
increased use of random effects models in preference to the current default of 
fixed effects models" (National Research Council, 1992, p. 185). 

Concretely speaking, the use of random-effects procedures in meta- 
analytic work will mean that in estimating the mean and its associated 
confidence interval, one will take into account not only the usual (human) 
sampling variation, but also the variability from one implementation to another 
( be tween-studies variance) . This has the effect of widening the confidence 
interval over what it would have been in a fixed-effects analysis (see Shadish 
& Haddock, 1994, p. 275; for related discussion, see Raudenbush, 1994, p. 306). 

Against this backdrop, I want to draw attention to the way in Which 
"Generalizing About Messages” invites consideration of the variability of 
effect as a phenomenon . With the increasing familiarity of meta-analytic work, 
we are now accustomed to thinking about effect size, and thus to 
conceptualizing a variable's effects as having some mean effect size. So, for 
example, in persuasion effects research — my own line of country, and a research 
domain affording easy examples — one can look across message replications and 
ask "what's the mean impact on persuasive outcomes across these 
instantiations?" 

But "Generalizing About Messages" implicitly invites us to look not only 
at the mean of the effect sizes, but also at their variability. (This is 
invited by an emphasis on random-effects analyses of replicated factors, in 
Which the variability among the implementations figures significantly. ) 

Indeed, the variation among observed effect sizes may be at least as 
interesting as the average effect across them. 

I want to make explicit here the contrast with some alternative views, 
especially as represented in some images of meta-analytic procedure. Sometimes 
it is supposed that the point of meta-analytic research is the establishment of 
sets of homogeneous effect sizes [homogeneous in the sense that a test for 
heterogeneity in the set of effect sizes fails to achieve significance (that 
is, the null hypothesis — that the variance of the effect sizes in the 
population is zero — is not rejected)]. From such a perspective, heterogeneity 
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in a collection of effect sizes is something to be squeezed out by ever-finer 
effect-size categorization. 

But one might alternatively take heterogeneity to be a fact about the 
phenomenon. For example, two compliance techniques might have the same mean 
effect size, but differ considerably in the degree of variability to be 
expected across implementations. That difference (in variability) might be 
interesting in and of itself. And certainly that difference could have 
straightforward practical implications: a persuader contemplating using one of 

these techniques would have a much better basis for predicting the likely 
effect in one case than in the other. In any event, the general point is that 
effect size variability — like the mean effect — can be thought of as simply one 
aspect of the phenomenon. 

The issues raised in "Generalizing About Messages" lead naturally to such 
thoughts. An emphasis on messages replications, coupled with the use of 
random-effects analyses, naturally draws attention to variability as a natural 
property of communication phenomena. 

So what I want to underscore today is the way in which "Generalizing 
About Messages" displays the interplay of substantive and methodological issues 
in research. It displays this not only by virtue of its procedural 
recommendations having been stimulated by a consideration of the substantive 
character of the phenomena under study, but also by virtue of the capacity that 
its procedural recommendations have to shape our conception of the phenomena of 
interest . 

I don't mean to say that all methodological innovations have the sort of 
substantive connections that "Generalizing About Messages" did. On the 
contrary, it's a hallmark of the most significant methodological contributions 
that they turn out to have the capacity to alter not only our customary 
research procedures, but also our very conception of the phenomena under 
study — all the more reason, then, that "Generalizing About Messages" is a 
deserving recipient of the Charles Woolbert award. 




6 



< ✓ 



References 

Brashers, D. (1996, November). The Impact of "Generalizing About 
Messages" on communication research. Paper presented at the annual convention 
of the Speech Communication Association, San Diego, CA. 

Dillard, J. P. , Hunter, J. E., & Burgoon, M. (1983, May). A meta- 
analysis of two sequential-request strategies for gaining compliance: Foot-in- 

the-door and door-in- the-f ace . Paper presented at the annual meeting of the 
International Communication Association, Dallas, TX. 

Hunter, J. E., Schmidt, F. L. , & Jackson, G. B. (1982). Meta-analysis : 
Cumulating research findings across studies . Beverly Hills, CA: Sage. 

Jackson, S. (1992). Message effects research: Principles of design and 

analysis . New York: Guilford. 

Jackson, S., & Jacobs, S. (1983). Generalizing about messages: 
Suggestions for design and analysis of experiments. Human Communication 
Research, 9 , 169-181. 

National Research Council (1992). Combining information: Statistical 

issues and opportunities for research . Washington, DC: National Academy 

Press. 

Rauderibush, S. W. (1994). Random effects models. In H. Cooper & L. V. 
Hedges (Eds.), Handbook of research synthesis (pp. 301-321). New York: 

Russell Sage Foundation. 

Shadish, W. R. , & Haddock, C. K. (1994). Combining estimates of effect 
size. In H. Cooper & L. V. Hedges (Eds.), Handbook of research synthesis (pp. 
261-281). New York: Russell Sage Foundation. 




8 



Would you like to put your paper in ERIC? Please send us a dark, clean copy! 







U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
Educational Resources Information Center (ERIC) 



I. DOCUMENT IDENTIFICATION: 



REPRODUCTION RELEASE 

(Specific Document) 



Title: p a per presented at the Annual Meeting of the Speech Communication Association (1996) 



Corporate Source: 



1 Publication Date: 



II. REPRODUCTION RELEASE: 

In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced 
in the monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced 
paper copy, and electronic/optical media, and sold through the ERIC Document Reproduction Service (EDRS) or other ERIC vendors. Credit is 
given to the source of each document, and, if reproduction release is granted, one of the following notices is affixed to the document 

If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following two options and sign at 
the bottom of the page. 



q 

t 



The sample sticker shown below will be 
affixed to all Level 1 documents 



The sample sticker shown below will be 
affixed to all Level 2 documents 



Check here 
For Level 1 Release: 

Permitting reproduction in 
microfiche (4* x 6" film) or 
other ERIC archival media 
(e.g., electronic or optical) 
and paper copy. 



PERMISSION TO REPRODUCE AND 




PERMISSION TO REPRODUCE AND 


DISSEMINATE THIS MATERIAL 




DISSEMINATE THIS 


HAS BEEN GRANTED BY 




MATERIAL IN OTHER THAN PAPER 






COPY HAS BEEN GRANTED BY 


V G 










Jr 


j 






TO THE EDUCATIONAL RESOURCES 




TO THE EDUCATIONAL RESOURCES 


INFORMATION CENTER (ERIC) 




INFORMATION CENTER (ERIC) 


Level 1 


Level 2 



□ 



Check here 
For Level 2 Release: 

Permitting reproduction in 
microfiche (4* x 6* film) or 
other ERIC archival media 
(e.g., electronic or optical), 
but not in paper copy. 



Documents will be processed as indicated provided reproduction quality permits. If permission 
to reproduce is granted, but neither box is checked, documents will be processed at Level 1. 



Sign 

here-* 

please 



O 

ERIC 



"/ hereby grant to the Educational Resources information Center (ERIC) nonexclusive permission to reproduce and disseminate 
this document as indicated above. Reproduction from the ERIC microfiche or electronic/optical media by persons other than 
ERIC employees and its system contractors requires permission from the copyright holder. Exception is made for non profit 
reproduction by libraries and other service agencies to satisfy information needs of educators in response to discrete inquiries . 9 



Signature: \ Printed Name/Position/Title: 


Organization/Address: ' j Telephone: 

m . r - _ 1 1 v : .* 


FAX: 

1A 7--Z44- \€°l& 


CwA o+ : E-Mail Address: 

l) rn_ 61&0I-3LR4- jclokeek^UiUd.e^ 


Date: 

TrjLfytl IW 



fnvar) 




III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce Is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, 
please provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is 
publicly available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are 
significantly more stringent for documents that cannot be made available through EDRS.) 



Publisher/Distributor: 




Address: 




Price: 





IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant reproduction release is held by someone other than the addressee, please provide the appropriate name and address: 



Name: 



Address: 



V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clearinghouse: 



ERIC/REC 
2805 E. Tenth Street 
Smith Research Center, 150 
Indiana University 
Bloomington, IN 47408 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 



' O 3/96) 

ERIC 



E R I C Pr o c e ccing ond Rofcrenoo Faci li ty 
*11 00 W e nt Str ee t , ?rf Hoot 
-fa aure l , Mai y land 20707 =35 98 

T elep h o n e : 001 -4 07 - 400 0 
- To l l F ree: 000 700 37 4 2 
FAX! 3 0 1 053 -03 6 3 

n nrn i l i nrlrfoi*0inat oH 

WWW: littp J/ui icfac.pieeord.oco.cona 



